Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Rework ObjImporter to support more features #205

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

Squareys
Copy link
Contributor

@Squareys Squareys commented Jun 8, 2017

Hello everybody!

I started reimplementing the ObjImporter pretty much a year ago for a University project, since I needed to import .obj files with materials.

While back then I only needed to support the features that I needed for those specfic .obj files, I decided to try to also implement support for all the previously supported features to hopefully get this into upstream someday.

Added features include:

  • Materials (usemtl and mtllib keywords)
  • Groups (g keyword)
  • Mixed primitive meshes
  • Zero-copy parsing (almost at least?)

Currently alot of tests are failing. The new implementation seems to be especially bad at failing for bad files still. There is quite alot to do still and may take another couple of days for me to get into a reviewable state.

Cheers, Jonathan.

TODOs

  • Add file callbacks feature

@Squareys
Copy link
Contributor Author

Squareys commented Jun 8, 2017

Oh, I guess I should proabably rebase the branch sometime again :D

  • done

Copy link
Owner

@mosra mosra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, first an apology from my side -- my original code was quite terrible so it's hard to improve it. I hope I didn't put you off with those 36 comments, though -- sorry about that.

std::unique_ptr<File> _file;
Containers::Array<char> _in;
std::unique_ptr<struct ImporterState> _state;
std::string _fileRoot;
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd vote for keeping just the single unique_ptr containing all the state, maybe even keeping the original File struct forward-declared above. Easier to clean up on close, less includes in the header.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✔️ done

@@ -28,81 +28,338 @@
#include <fstream>
#include <limits>
#include <sstream>
#include <tuple>
#include <stdlib.h>
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

<cstdlib> .. why is it needed, by the way?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that was because of strtol and strtof

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✔️ done


#include "MagnumPlugins/TgaImporter/TgaImporter.h"

using namespace Corrade::Containers;
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please no using namespace.

Copy link
Contributor Author

@Squareys Squareys Jun 9, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 But it makes alot of things more convenient... Aaaand it's a "private" file? Obviously explicitness is often nicer, but is there any other reason for this?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not used on that many places ;)

My reasoning: if a namespace is annoyingly long to type, it indicates a problem with the API naming -- it should be shortened. Using using namespace would somewhat hide that problem, but not solve it, and also makes the code a bit more opaque for people that don't know what belongs to the used namespace. I would say that in this case the namespace is still bearable ;)

ObjMesh* faces = nullptr;

ObjMeshData()=delete;
ObjMeshData(ObjGroup& g, int matId): group(g), materialId(matId) {}
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to delete default constructors if there are other constructors defined. Also, would make the other one explicit. Similarly below.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✔️ done

}

// find next non-whitespace char and return suffix at that point.
// param newlineIsWhitespace if "true", '\n' and '\r' are skipped also
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coding style: please use /* */-style comments, the // ones are treated as temporary to-be-removed comments.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know! :D It is a temporary comment, so I don't forget to put decent wording for other people there.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✔️ done (Cleanly formulated all the helper documentations and checked for Captial letter at comment starts everywhere)

CORRADE_COMPARE(importer.mesh3DCount(), 1);
}

void ObjImporterTest::unnamedMesh() {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this removed? Then there's no test case that tests that mesh3DForName("") returns -1

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, for the empty file, we now don't load anything at all...
I could put a mesh there, and then check the mesh3DForName(""). But why should it be returning -1 anyway?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see. Would be nice to have the check for both unnamed object and mesh then.

-1 is because there can be multiple meshes with empty name -- so you can map their ID to an empty name but not the other way around.

@@ -83,6 +82,8 @@ struct ObjImporterTest: TestSuite::Tester {
void missingNormalIndices();
void missingTextureCoordinateIndices();

void multiMaterialObject();

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a test case for the actual material data import?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point.

}

return MeshData3D{*primitive, std::move(indices), {std::move(positions)}, std::move(normals), std::move(textureCoordinates), {}, nullptr};
return std::unique_ptr<AbstractMaterialData>(mat);
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the material would be fully parsed in this function instead of in parse(), you wouldn't need the temporary objMat structure I think.

I hope I'm not talking about impossible things here, though ;)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I guess it would be enough to only parse the material names in parse() and maybe store their location in ArrayView<const char>s instead

/* Comment line */
if(pos[0] == '#') {
pos = ignoreLine(pos);
pos = skipWhitespaces(pos);
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, this is done right in the next loop iteration, no need to do this here as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, no, the ignore Whitespaces ignores words before the keyword (nextWord() expects first character to be non-whitespace, otherwise will return empty string, indicating there is no word at the beginning), versus the other one skips whitespaces after the keyword.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe Wikipedia has it wrong, but I saw some prefix whitespace in the code samples there: https://en.wikipedia.org/wiki/Wavefront_.obj_file#File_format

I personally don't see why there should be such a restriction of no whitespace at line beginning.

Copy link
Contributor Author

@Squareys Squareys Jun 9, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I never said it was a restriction, the parser will parse ······\t····p·1 just fine with my current implementation?

Ah! The point is that I ignore those newlines before using nextWord(). Hence the input to that will always be p·1 in the example above, because the others have already been ignored.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the parser will parse ······\t····p·1 just fine with my current implementation?

My point is that this wouldn't work if this would be the very first line (as far as I can see), so I'm suggesting putting the skipWhitespaces() as the very first thing in the loop, instead of very last ;)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, alright, yeah, I seem to have removed the first skipWhitespace call somewhere, but what you say makes sense

/* Create dummy mesh for this object so that it gets loaded as ObjectData */
_state->meshlessObjects.push_back(object->name);
}
};
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit unsure about these inline lambdas -- i'm personally always trying to do such cleanup code only once in the parsing loop, because otherwise the corner-case testing gets quite out of hand. But I didn't stare at the code long enough yet to see how that could be done :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will not be easy, will stare at the code for a while and tell you what I think after.

@Squareys
Copy link
Contributor Author

Squareys commented Jun 9, 2017

Thanks for the initial review!

I hope I didn't put you off with those 36 comments, though -- sorry about that.

Never! I am always happy to learn! :D

A couple of things were pretty obvious, though, these things would have gotten resolved in an autonomous code review also 😉 I don't think commenting on code stlye/formatting makes too much sense yet, since I didn't do that final cleanup pass myself yet.

I will have time to apply the requested changes over the weekend :)

/* Not using PhongMaterialData, since we may want to set color and texture to
* later decide which flags we set. We do not know whether we have a texture
* or color beforehand. */
struct ObjMaterial {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, forgot to say: could the internal Obj* structs be wrapped in an anonymous namespace as well to prevent symbol leakages? Thank you.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✔️ done



/* The state of the imported generated by openData() */
struct ImporterState {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for putting it all in one -- could it be ObjImporter::State or something so it doesn't appear in the global Trade namespace? There is a leftover struct File forward-declaration in the header also, so you could just reuse that line.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously the state unique_ptr was reset in doOpen*(), but those two added members in and fileRoot shouldn't be, because they are set before the state is reset. Hence two options:
Either unique_ptr<File> with a new struct File in the Importer state struct (which can be reset independently from the importer state) or revert back to having the two members outside of this struct.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took me a while to realize what's going on, sorry for the late reply.

but those two added members in and fileRoot shouldn't be, because they are set before the state is reset

I think they don't need to be. One option:

void ObjImporter::doOpenFile(const std::string& filename) {
    AbstractImporter::doOpenFile(filename);
    _state->fileRoot = Utility::Directory::path(filename);
}

void ObjImporter::doOpenData(Containers::ArrayView<const char> data) {
    _state.reset(new ImporterState);
    _state->in = Containers::Array<char>{data.size()};
    std::copy(data.begin(), data.end(), _state->in.begin());
    parse();
}

The above assumes that parse() doesn't need to access the _fileRoot parameter (which it shouldn't, if you defer the material parsing to later, as I suggested). Another option could be this, where the _state is only populated if it's not already there:

void ObjImporter::doOpenFile(const std::string& filename) {
    _state.reset(new ImporterState);
    _state->fileRoot = Utility::Directory::path(filename);
    AbstractImporter::doOpenFile(filename);
}

void ObjImporter::doOpenData(Containers::ArrayView<const char> data) {
    if(!_state) _state.reset(new ImporterState);
    _state->in = Containers::Array<char>{data.size()};
    std::copy(data.begin(), data.end(), _state->in.begin());
    parse();
}

The AbstractImporter::open*() function implementation is unconditionally calling close() before calling the doOpen() implementations, so you can be always sure that if _state is already populated, it's yours and not some stale one from a previous file.

@sariug
Copy link

sariug commented Apr 16, 2019

Hello,

I am trying to open an obj file. Tried it with 3 different version of it. For same obj file:

  • I tried opening the original
  • Opened in MeshLab and saved again with Vertex+color information
  • Opened in MeshLab and saved again without Vertex+color information

None of the worked nicely with Obj viewer plugin.
Errors I have gotten are, respectively:
-Trade::ObjImporter::mesh3D(): error while converting numeric data
-Trade::ObjImporter::mesh3D(): invalid float array size
-Opening file deneme2.obj; Importing mesh 0 ; Cannot load the mesh, skipping

Do you have any other suggestions to handle these issue as well as is there any expected improvement in this plugin ? or any other suggestions to deal with it ?

The obj file looks like :
image

Regards,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In progress
Development

Successfully merging this pull request may close these issues.

3 participants